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BACKGROUND 


In 2014, the U S Department of Education awarded the Maricopa County Education Service 
Agency an Investing in Innovation (i3) grant to fund the Engineering STEM Identity (ESI) 
project. The purpose of i3 is to expand the implementation of, and investment in, innovative 
educational practices. Accordingly, the goal of ESI is to address one of the most important 
challenges facing education: developing student interest and proficiency in the fields of Science, 
Technology, Engineering, and Mathematics (STEM). 

As part of the evaluation of i3 funded projects, grantees must study and report on the degree to 
which a project is carried out as planned. This type of study provides researchers and 
practitioners a better understanding of how and why an intervention works, and the extent to 
which outcomes can be improved. Importantly, if it is found that the project was not 
implemented properly then it is not possible to clearly conclude that the project had an effect on 
the students’ performance. 

The Fidelity of Implementation (FOI) document presents a conceptual and analytic framework 
for assessing the quality of the implementation of the ESI project. The conceptualization of the 
present FOI was completed October 1, 2014 after reviewing the ESI grant activities and selected 
quantitative ratings to be employed. The approved FOI approach assesses the implementation of 
three constructs: 

Construct 1: Curriculum and Instruction. This is defined by the extent to which participating 
teachers implement four key indicators (components) of the ESI program: 

■ Indicator 1.1: Modeling instruction 

■ Indicator 1.2: STEM Pro Spotlights 

■ Indicator 1.3: Challenge Cohorts 

■ Indicator 1.4: Peer Panels 

Construct 2: Teacher Professional Development. Three indicators are studied: 

■ Summer Institute Modeling Curriculum 

■ STEM Coaching 

■ Leading Challenge Cohorts Discussions 

Construct 3: Administrative Support of the principal or other administrative personnel. This 
is measured by studying one indicator: 

■ Principal and school district Central Office support 

Each Construct requires three levels of evaluation: 

1. The extent to which each participating teacher (Constructs 1 and 2) or administrative 
team (Construct 3) properly implemented a specific indicator. 

2. An assessment of the extent to which each teacher or administrative team, in the case of 
Construct 3, implemented all of the Construct related indicators. 

3. A detennination of whether the program taken as a whole implemented the Construct. 

The following report evaluates the fidelity of implementation for each of the three (3) Constructs and 
their associated indicators. A detailed presentation of the evaluation is provided in the Appendix. 
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FINDINGS 


Construct 1. Curriculum and Instruction 

Construct 1 is measured by the four indicators described below. For each fidelity measure there 
is a threshold that must be met or exceeded to result in a finding of “meeting or exceeding”. 

1.1: Modeling Instruction. Adequate fidelity by a teacher is defined as: 

Scoring twenty-two (22) points or more based on classroom observation by STEM Coach using 
Modeling Implementation Checklist. 

1.2: Stem Pro Spotlights. Adequate fidelity by a teacher is defined as: 

Scoring eleven (11) or more points based on observation by STEM PCK Expert using ESI 
STEM Pro Spotlight Observation Protocol. The STEM Coach and PCK Expert conduct all 
interactions. 

1.3: Challenge Cohorts. Adequate fidelity by a teacher is defined as: 

Scoring fifteen (15) or more points based on observation by STEM PCK Expert using ESI 
Challenge Cohort Observation Protocol. 

1.4: Peer Panels. Adequate fidelity by a teacher is defined as: 

Scoring nine (9) points or more based on observation by STEM Coach using Peer Panel Meeting 
Observation Log. 

Each of the Construct 1 indicators was analyzed to determine if teachers met or exceeded the 
fidelity standards. The results are displayed in Table 1. 


Table 1. Fidelity of Indicators of Construct 1 


Indicator 

Percent of Teachers 
Meeting or Exceeding 
Fidelity 

Standard Met 

1.1 Modeling Instruction 

86% 

Yes 

1.2 STEM Pro Spotlights 

100% 

Yes 

1.3 Challenge Cohorts 

93% 

Yes 

1.4 Peer Panels 

100% 

Yes 
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In addition to determining if teachers met or exceeded the standard established for each 
indicator, the ESI is required to assess the fidelity of implementation for Construct 1: Curriculum 
and Instruction by studying: A) the Construct Level Fidelity for each Teacher and B) the 
Construct Level Fidelity for the Program. 

A. Construct Level Fidelity for Teachers: 

Teachers will meet fidelity standards by receiving top scores of 4 or 5 on indicators which mean 
the teachers met or exceeded fidelity standards. A teacher meets or exceeds adequate fidelity 
threshold for at least 3 of the 4 indicators ( Indicators 1.1 - 4 .). 

Findings: 100% of the teachers met or exceeded the fidelity indicators. Accordingly, 

the standard for the Construct Level Fidelity for Teachers was met. 


B. Construct Fidelity for Program: 

Program will meet fidelity standards if 75% of the teachers will meet or exceed fidelity standards 
on three of the four indicators ( Indicators 1.1-4 .). 

Findings: 95% of the teachers met this threshold of Construct 1 Fidelity. 

Accordingly, the standard for the Construct Fidelity for Program was met. 


C. Overall: Construct 1 Curriculum and Instruction: 

Findings: Met Fidelity of Implementation standards or objectives. 
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Construct 2. Teacher Professional Development 

Construct 2 is measured by the three indicators described below. For each fidelity measure there 
is a threshold that must be met or exceeded to result in a finding of “meeting or exceeding”. 

2.1: Summer Institute Modeling Curriculum. Adequate fidelity is defined as: 

100% of the teachers will participate in the 15 days of summer training onsite or offsite through 
differentiated training throughout the year. 

2.2: STEM Coaching. Adequate fidelity is defined as: 

75% of the teachers will have been determined to be at the Refinement (4) or the Integration/ 
Renewal (5) levels based on STEM Coaches observation and review. 

2.3: Leading Challenge Cohorts Discussions: Adequate fidelity is defined as: 

75% of the teachers will be at the Refinement (4) or the Integration/ Renewal (5) levels based on 
STEM experts assessments and their meetings with individual teachers at which time they 
review and develop challenge designs. 

Each of the Construct 2 indicators was analyzed to determine if teachers met or exceeded the 
fidelity standards. The results are displayed in Table 2. 


Table 2. Attendance at Summer Modeling Institute Indicator 2.1 


Number Teachers 

Onsite/Offsite 

Attendance and 
Offsite Training 

Attendance 

2 

Onsite/Offsite 

100% 

5 

1 

Onsite/Offsite 

100% 

12 

1 

Onsite/Offsite 

100% 

13 

9 

Onsite/Offsite 

100% 

14 

6 

Onsite 

100% 

15 

1 

Offsite 

100% 

0 

Total 20 


100% 



Note . One Science teacher did not attend the Institute but received the Institute differentiated 
training completely offsite via. 

Findings: A total of 100% of the 20 teachers attended all fifteen days of the Institute or 
received differentiate training throughout the year. 
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Table 3. Teachers Meeting Fidelity Indicators for Construct 2 


Indicator 

Percent of Teachers 
Meeting or Exceeding 
Fidelity 

Standard Met 

2.1: Summer Institute Modeling 
Curriculum 

100% 

Yes 

2.2: STEM Coaching 

85% 

Yes 

2.3: Leading Challenge Cohorts 
Discussions 

85% 

Yes 


In addition to determining if teachers met or exceeded the standard established for the indicator, 
the ESI is required to assess the overall fidelity of implementation for Construct 2: Teacher 
Professional Development by studying: A) the Construct Level Fidelity for each Teacher and B) 
Construct Level Fidelity for the Program. 

A. Construct Level Fidelity for Teachers: 

Teachers will meet fidelity standards: if they have been determined to have met or exceeded 
adequate fidelity threshold for both indicators 2.2 (STEM Coaching) and 2.3 (Leading Challenge 
Cohorts Discussions). 

Findings: 85% of the teachers met or exceeded adequate fidelity threshold for both 

indicators 2.2 and 2.3. Standard met. 


B. Construct Fidelity for Program: 

The ESI Program will meet fidelity standards if 100% of the teachers will participate in the ESI 
Summer Institute or receive the Institute training through differentiated training throughout the 
year. (Indicator 2.1) AND at least 75% of the teachers have a score of “1” for the Teacher 
Professional Development construct (Teachers meet adequate fidelity threshold for both 
indicators 2.2 (STEM Coaching) and 2.3 (Leading Challenge Cohorts Discussions). 

Findings: 100% institute participation and 85% of the teachers met the standards for 

indicators 2.2 and 2.3. Accordingly, the standard for Construct 2, Fidelity 
for the Program was met. 


C. Overall: Construct 2: Professional Development: 

Findings: Met Fidelity of Implementation standards or objectives. 
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Construct 3. Administrative Support 

Construct 3 is measured by one indicator. The threshold for this indicator must be met or 
exceeded to result in a finding of “meeting or exceeding” the threshold. 

3.1: Principal and administrative personnel support. Each principal/school will be rated on a 
1-30 point scale using the Principal Support Checklist that measures live areas: 1) overall school 
environment, 2) obtaining/leveraging resources to support STEM, 3) principal/ central office 
participating in STEM activities, 4) establishing time for STEM teachers to collaborate with 
classroom teachers, and 5) actively expand STEM curriculum and instruction. 

Adequate fidelity by a principal/school is defined as scoring 20 or more points. 


Table 4. Principals/Schools Meeting Fidelity Indicator 3.1 


Indicator 

Percent Meeting or 
Exceeding Fidelity 

Standard Met 

3.1: Principal and school district 
Central Office support 

94% 

Yes 


In addition to determining if Principals / Schools met or exceeded the standard established for 
the indicator, the ESI is required to assess the overall fidelity of implementation for Construct 3: 
Administrative Support by studying : A) Construct Level Fidelity for principals/schools and B) 
Construct Level Fidelity for the Program. 

A. Construct Level Fidelity for Principals/Schools: 

A principal/school will met or exceeded the threshold for adequate fidelity if the principal/school 
scored 20 or more points in at least 2 of the 3 reporting periods per school year. 

Findings: 97% of principals met the fidelity standard. Standard met. 

B. Construct Fidelity for Program: 

The ESI Program will meet fidelity standards if 75% of the principals demonstrate administrative 
support during 2 of the 3 reporting periods. 

Findings: 97% of principals met the fidelity standard. Standard met. 

C. Overall: Construct 3 Administrative Support 

Findings: Met Fidelity of Implementation standards or objectives. 
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SUMMARY 


Construct 1 Curriculum and Instruction : 

The Level of Fidelity of Implementation of the teachers will meet or exceed standards (100%), 
and 95% of the teachers met the Fidelity of Program implementation. 

Determination: Fidelity standards of teachers and program were met. 


Construct 2 Teacher Professional Development : 

Level of Fidelity of Implementation of teacher professional development in STEM Coaching and 
Leading Challenge Cohorts Discussion was met by 85% of the teachers. 

Determination: Fidelity Standards for teachers were met. 

Construct 2 Level of Fidelity for the Program required 100% participation at the Summer 
Modeling Institute by attendance or differentiated training and at least 75% of the teachers 
meeting the STEM Coaching and Leading Challenge Cohorts Discussion thresholds. The 
thresholds were met in the STEM Coaching and Leading Challenge Cohorts Discussion 
indicators. 

Determination: Fidelity standards were met. 

Construct 3 Administrative Support : 

The Level of Fidelity of principal/schools supporting the grant met by 97% of the principals. 
Determination: Fidelity Standards for principals/schools were met. 


APPENDIX. 

The Appendix displays the rating scales for the constructs of the Fidelity of Implementation 
Report. These tables provide insights to the rating process. 
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APPENDIX 


Table 5. Means and Percentages for Indicator 1.1. 


Indicator 

N 

M 

Percent Meeting or 
Exceeding Standards 

1. Lesson demonstrated teacher's content 
knowledge 

22 

3.95 

86.4% 

2. Students' prior knowledge and 
preconceptions were identified by teacher 
through instruction 

22 

3.45 

59.1% 

3. Students participated actively in group 
based lab investigations 

22 

3.64 

72.7% 

4. Task-related student discourse occurred 
independent of teacher 

22 

3.82 

63.6% 

5. Model(s) were represented in multiple 
ways (verbal, data, graphical, mathematical, 
diagrammatically, etc.) 

22 

2.95 

31.8% 

6. Vernier (data gathering) technology was 
used during lesson to develop model(s) 

22 

3.91 

63.6% 

7. Teacher used GoClass program often 
during the lesson 

22 

1.95 

13.6% 


Note. N refers to the total number of teachers observed during Quarters B, C, and D. The percent 
meeting or exceeding standards represents the number of participants scoring 4 or 5 points. 
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Table 6. Fidelity Score Benchmarks: Indicator 1.1. 


No 

Implementation 
(0-7 points) 

Partial 

Implementation 
(8 - 14 points) 

Inadequate 
Implementation 
(15 - 21 points) 

Meets 

Standards for 
Implementation 
(22 - 28 points) 

Exceeds 
Standards 
(29 - 35 points) 

1 (4.5%) 

1 (4.5%) 

1 (4.5%) 

16 (72.7%) 

3 (13.6%) 

Total Inadequate Fidelity Score Total Adequate Fidelity Score 

(21 points or below) 


(22 points or more) 


3 (13.6%) 


19 (86.4%) 


Note. N = 22. 
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Table 7. Means and Percentages for Indicator 1.2. 


Indicator 

N 

M 

Percent Meeting or 
Exceeding Standards 

1. STEM Pro engages students in becoming 
excited about potential STEM fields and/or 
careers 

25 

4.52 

100.0% 

2. STEM Pro provides opportunities for 
students to relate to him/her by sharing 
whole identity 

25 

4.44 

88.0% 

3. Teacher facilitates students in a way that 
sets students and STEM Pro up for 
successful interactions 

25 

3.60 

60.0% 

4. Students are consistently well behaved 
and on-task 

25 

4.20 

80.0% 

5. Students are engaged in conversation 
with STEM Pro as evidenced by active 
participation 

25 

3.48 

52.0% 

6. Technology functioned in a way that 
supported STEM Pro, teacher, and student 
interactions 

25 

4.92 

100% 


Note. N refers to the total number of teachers observed during Quarters B, C, and D. The percent 
meeting or exceeding standards represents the number of participants scoring 4 or 5 points. 
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Table 8. Fidelity Score Benchmarks: Indicator 1.2 


No 

Implementation 
(0-3 points) 

Unacceptable 
Implementation 
(4-6 points) 

Partial 

Implementation 
(7 -10 points) 

Meets 

Standards for 
Implementation 
(11-25 points) 

Exceeds 
Standards 
(26 - 30 
points) 

0 (0.0%) 

0 (0.0%) 

0 (0.0%) 

12 (48.0%) 

13 (52.0%) 

Total Inadequate Fidelity Score Total Adequate Fidelity Score 

(10 points or below) (11 points or more) 


0 (0.0%) 


25 (100.0%) 



Note. N=25. 
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Table 9. Means and Percentages for Indicator 1.3 


Indicator 

N 

M 

Percent Meeting or 
Exceeding Standards 

1. STEM professional is able to provide 
assistance in the development of the 
Engineering Challenge by acting as 
consultant to individualized classroom 
needs 

14 

1.21 

7.1% 

2. STEM professional is able to provide 
assistance in the implementation of the 
Engineering Challenge by acting as 
consultant to individualized classroom 
needs 

14 

1.21 

7.1% 

3. Students and teachers create an 
engineering challenge that is specific to the 
needs of their school; students perceive that 
the identified challenge was determined by 
themselves 

14 

3.86 

71.4% 

4. PCK Expert is able to mediate 
conversation between STEM Pro, 
classrooms, and teachers 

14 

4.71 

92.9% 

5. Students are consistently well behaved 
and on-task during all interactions with the 
STEM Pro 

14 

4.00 

78.6% 

6. The IVL technology functions in a way 
that supports STEM Pro, PCK Expert, 
teachers, and student interactions 

14 

4.93 

100.0% 


Note. N refers to the total number of teachers observed during Quarters C, and D. The percent 
meeting or exceeding standards represents the number of participants scoring 4 or 5 points. 
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Table 10. Fidelity Score Benchmarks: Indicator 1.3 


No 

Implementation 
(0-5 points) 

Unacceptable 
Implementation 
(6 -10 points) 

Partial 

Implementation 
(11-14 points) 

Meets 

Standards for 
Implementation 
(15 - 26 points) 

Exceeds 
Standards 
(27 - 30 
points) 

0 (0.0%) 

1 (6.25%) 

0 (0.0%) 

13 (93.75%) 

0 (0.0%) 

Total Inadequate Fidelity Score Total Adequate Fidelity Score 

(14 points or below) 


(15 points or more) 


1 (7.1%) 


13 (92.9%) 


Note. N = 14. 


Table 11. Means and Percentages for Indicator 1.4 


Indicator 

N 

M 

Percent Meeting or 
Exceeding Standards 

1. Teachers managed class behavior and 
interactions effectively to advance student 
learning 

26 

3.62 

53.8% 

2. Students demonstrated excitement and 
enthusiasm while engaging 

26 

3.62 

61.5% 

3. Meeting content was relevant and 
integrated with class curriculum 

26 

3.77 

69.2% 

4. Student conversations between 
classrooms occurred via the IVL technology 

26 

3.58 

53.8% 

5. The IVL technology functioned to 
support class meeting 

26 

3.73 

53.8% 


Note. N refers to the total number of teacher pairs observed during Quarters B, C, and D. The 
percent meeting or exceeding standards represents the number of participants scoring 4 or 5 
points. 
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Table 12. Fidelity Score Benchmarks: Indicator 1.4 


No 

Implementation 
(0-3 points) 

Unacceptable 
Implementation 
(4-6 points) 

Partial 

Implementation 
(7-8 points) 

Meets 

Standards for 
Implementation 
(9 - 22 points) 

Exceeds 
Standards 
(23 - 25 
points) 

0 (0.0%) 

0 (0.0%) 

0 (0.0%) 

23 (88.5%) 

3 (11.5%) 

Total Inadequate Fidelity Score Total Adequate Fidelity Score 

(8 points or below) 


(9 points or more) 


0 (0.0%) 


26 (100.0%) 



Note. N = 26. 


Table 13. Fidelity Score Benchmark: Indicator 2.2 


Nonuse/Basic 
Orientation 
(1 point) 

Mechanical 

Use 

(2 points) 

Routine Use 
(3 points) 

Refinement 
(4 points) 

Integration/Renewal 
(5 points) 

0 (0.0%) 

1 (5.0%) 

2(10.0%) 

11 (55.0%) 

6 (30.0%) 

Total Inadequate Fidelity Score 

Total Adequate Fidelity Score 

(3 points or below) 


(4 points or more) 

3 (15.0%) 


17 (85.0%) 


Note. N = 20. 
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Table 14. Fidelity Score Benchmark: Indicator 2.3 


Nonuse/Basic 
Orientation 
(1 point) 

Mechanical 

Use 

(2 points) 

Routine Use 
(3 points) 

Refinement 
(4 points) 

Integration/Renewal 
(5 points) 

0 (0.0%) 

2(10.0%) 

1 (5.0%) 

6 (30.0%) 

11 (55.0%) 

Total Inadequate Fidelity Score 
(3 points or below) 

Total Adequate Fidelity Score 
(4 points or more) 


3 (15.0%) 


17 (85.0%) 

Note. N = 20. 


Table 15. Means and Percentages for Indicator 3.1 

Indicator 

N 

M 

Percent Meeting or 
Exceeding Standards 

1. Overall school environment 

16 

4.16 

100.0% 

2. Resources are available to support ESI 

16 

4.38 

100.0% 

3. Principal / central office participates in 

ESI activities 

16 

4.25 

100.0% 

4. Establish time for ESI teachers to 
collaborate with other teachers 

16 

4.16 

90.63% 

5. Expand STEM curriculum and 
instruction 

16 

4.03 

90.63% 

6. Additional credit based on exceptional 
performance in one or more areas 

16 

1.38 

0.0% 


Note. N refers to the total number of observations conducted. The percent meeting or exceeding 
standards represents the number of participants scoring 4 or 5 points. Each of the 16 principals 
was assessed on two occasions. 
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Table 16. Fidelity Score Benchmark: Indicator 3.1 


No 

Implementation 
(1-6 points) 

Low 

Implementation 
(7 -13 points) 

Moderate 
Implementation 
(14 - 19 points) 

Meets 

Standards for 
Implementation 
(20 - 25 points) 

Exceeds 
Standards 
(26 - 30 
points) 

0 (0.0%) 

0 (0.0%) 

1 (0.0%) 

27 (93.75%) 

4 (0.0%) 

Total Inadequate Fidelity Score Total Adequate Fidelity Score 

(19 points or below) 


(20 points or more) 


1 (3.13%) 


31 (96.9%) 



Note. 7V= 16 principals assessed on two occasions 
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